Publication bias in psychology: A closer look at the correlation between sample size and effect size.

Previously observed negative correlations between sample size and effect size (n-ES correlation) in psychological research have been interpreted as evidence for publication bias and related undesirable biases. Here, we present two studies aimed at better understanding to what extent negative n-ES correlations reflect such biases or might be explained by unproblematic adjustments of sample size to expected effect sizes. In Study 1, we analysed n-ES correlations in 150 meta-analyses from cognitive, organizational, and social psychology and in 57 multiple replications, which are free from relevant biases. In Study 2, we used a random sample of 160 psychology papers to compare the n-ES correlation for effects that are central to these papers and effects selected at random from these papers. n-ES correlations proved inconspicuous in meta-analyses. In line with previous research, they do not suggest that publication bias and related biases have a strong impact on meta-analyses in psychology. A much higher n-ES correlation emerged for publications' focal effects. To what extent this should be attributed to publication bias and related biases remains unclear.

Heterogeneity estimates in a biased world.

Meta-analyses typically quantify heterogeneity of results, thus providing information about the consistency of the investigated effect across studies. Numerous heterogeneity estimators have been devised. Past evaluations of their performance typically presumed lack of bias in the set of studies being meta-analysed, which is often unrealistic. The present study used computer simulations to evaluate five heterogeneity estimators under a range of research conditions broadly representative of meta-analyses in psychology, with the aim to assess the impact of biases in sets of primary studies on estimates of both mean effect size and heterogeneity in meta-analyses of continuous outcome measures. To this end, six orthogonal design factors were manipulated: Strength of publication bias; 1-tailed vs. 2-tailed publication bias; prevalence of p-hacking; true heterogeneity of the effect studied; true average size of the studied effect; and number of studies per meta-analysis. Our results showed that biases in sets of primary studies caused much greater problems for the estimation of effect size than for the estimation of heterogeneity. For the latter, estimation bias remained small or moderate under most circumstances. Effect size estimations remained virtually unaffected by the choice of heterogeneity estimator. For heterogeneity estimates, however, relevant differences emerged. For unbiased primary studies, the REML estimator and (to a lesser extent) the Paule-Mandel performed well in terms of bias and variance. In biased sets of primary studies however, the Paule-Mandel estimator performed poorly, whereas the DerSimonian-Laird estimator and (to a slightly lesser extent) the REML estimator performed well. The complexity of results notwithstanding, we suggest that the REML estimator remains a good choice for meta-analyses of continuous outcome measures across varied circumstances.

Heterogeneity of Research Results: A New Perspective From Which to Assess and Promote Progress in Psychological Science.

Heterogeneity emerges when multiple close or conceptual replications on the same subject produce results that vary more than expected from the sampling error. Here we argue that unexplained heterogeneity reflects a lack of coherence between the concepts applied and data observed and therefore a lack of understanding of the subject matter. Typical levels of heterogeneity thus offer a useful but neglected perspective on the levels of understanding achieved in psychological science. Focusing on continuous outcome variables, we surveyed heterogeneity in 150 meta-analyses from cognitive, organizational, and social psychology and 57 multiple close replications. Heterogeneity proved to be very high in meta-analyses, with powerful moderators being conspicuously absent. Population effects in the average meta-analysis vary from small to very large for reasons that are typically not understood. In contrast, heterogeneity was moderate in close replications. A newly identified relationship between heterogeneity and effect size allowed us to make predictions about expected heterogeneity levels. We discuss important implications for the formulation and evaluation of theories in psychology. On the basis of insights from the history and philosophy of science, we argue that the reduction of heterogeneity is important for progress in psychology and its practical applications, and we suggest changes to our collective research practice toward this end.

2D:4D Suggests a Role of Prenatal Testosterone in Gender Dysphoria.

Gender dysphoria (GD) reflects distress caused by incongruence between one's experienced gender identity and one's natal (assigned) gender. Previous studies suggest that high levels of prenatal testosterone (T) in natal females and low levels in natal males might contribute to GD. Here, we investigated if the 2D:4D digit ratio, a biomarker of prenatal T effects, is related to GD. We first report results from a large Iranian sample, comparing 2D:4D in 104 transwomen and 89 transmen against controls of the same natal sex. We found significantly lower (less masculine) 2D:4D in transwomen compared to control men. We then conducted random-effects meta-analyses of relevant studies including our own (k = 6, N = 925 for transwomen and k = 6, N = 757 for transmen). In line with the hypothesized prenatal T effects, transwomen showed significantly feminized 2D:4D (d ≈ 0.24). Conversely, transmen showed masculinized 2D:4D (d ≈ - 0.28); however, large unaccounted heterogeneity across studies emerged, which makes this effect less meaningful. These findings support the idea that high levels of prenatal T in natal females and low levels in natal males play a part in the etiology of GD. As we discuss, this adds to the evidence demonstrating the convergent validity of 2D:4D as a marker of prenatal T effects.

Visuo-spatial performance in autism: a meta-analysis.

Visuo-spatial skills are believed to be enhanced in autism spectrum disorders (ASDs). This meta-analysis tests the current state of evidence for Figure Disembedding, Block Design, Mental Rotation and Navon tasks in ASD and neurotypicals. Block Design (d = 0.32) and Figure Disembedding (d = 0.26) showed superior performance for ASD with large heterogeneity that is unaccounted for. No clear differences were found for Mental Rotation. ASD samples showed a stronger local processing preference for Navon tasks (d = 0.35); less clear evidence for performance differences of a similar magnitude emerged. We discuss the meta-analysis results together with other findings relating to visuo-spatial processing and three cognitive theories of ASD: Weak Central Coherence, Enhanced Perceptual Functioning and Extreme Male Brain theory.

No Evidence that 2D:4D is Related to the Number of CAG Repeats in the Androgen Receptor Gene.

The length ratio of the second to the fourth digit (2D:4D) is a putative marker of prenatal testosterone (T) effects. The number of CAG repeats (CAGn) in the AR gene is negatively correlated with T sensitivity in vitro. Results regarding the relationship between 2D:4D and CAGn are mixed but have featured prominently in arguments for and against the validity of 2D:4D. Here, I present random-effects meta-analyses on 14 relevant samples with altogether 1904 subjects. Results were homogeneous across studies. Even liberal estimates (upper limit of the 95% CI) were close to zero and therefore suggested no substantial relationship of CAGn with either right-hand 2D:4D, left-hand 2D:4D, or the difference between the two. However, closer analysis of the effects of CAGn on T dependent gene activation in vitro and of relationships between CAGn and T dependent phenotypic characteristics suggest that normal variability of CAGn has mostly no, very small, or inconsistent effects. Therefore, the lack of a clear association between CAGn and 2D:4D has no negative implications for the latter's validity as a marker of prenatal T effects.

Digit ratio (2D:4D) and male facial attractiveness: new data and a meta-analysis.

Digit ratio (2D:4D) appears to correlate negatively with prenatal testosterone (T) effects in humans. As T probably increases facial masculinity, which in turn might be positively related to male facial attractiveness, a number of studies have looked into the relationship between 2D:4D and male facial attractiveness, showing equivocal results. Here, I present the largest and third largest samples so far, which investigate the relationship between 2D:4D and male facial attractiveness in adolescents (n = 115) and young men (n = 80). I then present random-effects meta-analyses of the available data (seven to eight samples, overall n = 362 to 469). These showed small (r ≈ -.09), statistically non-significant relationships between 2D:4D measures and male facial attractiveness. Thus, 2D:4D studies offer no convincing evidence at present that prenatal T has a positive effect on male facial attractiveness. However, a consideration of confidence intervals shows that, at present, a theoretically meaningful relationship between 2D:4D and male facial attractiveness cannot be ruled out either.

Male body movements as possible cues to physical strength: a biomechanical analysis.

OBJECTIVES: Comparative research suggests that male courtship displays signal condition-dependent traits tofemales; these displays might also provide cues to potential male competitors. Although some associations betweenhuman movements and physical/behavioral qualities have been found, such research has typically only been conducted from a perspective of female mate choice. Here, using advanced motion capture and biomechanical analyses, we examine the extent to which male dancing provides cues about the dancer's physical qualities to both males and females. METHODS: Thirty men aged 19-37 were recorded using motion-capture technology as they danced to a standard rhythm. Participants also completed a vascular fitness test, assessments of upper- and lower-body strength, and biomechanical indices were extracted from their dance movements. Dance clips were converted into virtual humanoid characters (avatars) and rated by 27 women and 21 men on perceived dance quality. RESULTS: General linear mixed modeling revealed that both handgrip strength and arm movements of the dancer were statistically significant predictors of dance quality ratings; stronger males who displayed larger, more variable, and faster movements of their arms being rated as better dancers. There was no effect of the sex of the observer in predicting dance quality ratings, indicating that male and female observers rated dance quality equivalently. Physical fitness was not associated with perceived dance quality. CONCLUSIONS: Men and women are able to derive certain quality cues from observing male dance movements in the form of controlled stimuli. Thus, male dancing may form a condition-dependent ornament of certain aspects of mate quality.

Digit ratio 2D:4D in relation to autism spectrum disorders, empathizing, and systemizing: a quantitative review.

Prenatal testosterone (PT) effects have been proposed to increase systemizing (the drive to understand lawful input-output relationships), to decrease empathizing (the drive to understand others), and to cause autism via hypermasculinization of the brain. Digit ratio 2D:4D is a putative marker of PT effects in humans. An online study (n = 1896) into the relationship between the Reading the Mind in the Eyes Test (a widely used measure of empathizing) and self-measured 2D:4D in a nonclinical sample is reported. No evidence for a link between empathizing and 2D:4D in either females or males emerged. Further, three meta-analyses are presented that look into the relationships of 2D:4D with autism spectrum disorder (ASD), systemizing, and empathizing. 2D:4D was substantially lower (more masculine) in ASD-affected individuals than in normal controls (d = -0.58, P < 0.001). However, 2D:4D was found to be virtually unrelated to systemizing and empathizing in normal adults. The results support the idea that high PT is a risk factor for autism, but they challenge the view that PT substantially contributes to sex differences in systemizing and empathizing. Possibly, this pattern reflects an interaction effect, whereby PT drives ASD characteristic changes only in brains with a specific damage.

Male dance moves that catch a woman's eye.

Male movements serve as courtship signals in many animal species, and may honestly reflect the genotypic and/or phenotypic quality of the individual. Attractive human dance moves, particularly those of males, have been reported to show associations with measures of physical strength, prenatal androgenization and symmetry. Here we use advanced three-dimensional motion-capture technology to identify possible biomechanical differences between women's perceptions of 'good' and 'bad' male dancers. Nineteen males were recorded using the 'Vicon' motion-capture system while dancing to a basic rhythm; controlled stimuli in the form of avatars were then created in the form of 15 s video clips, and rated by 39 females for dance quality. Initial analyses showed that 11 movement variables were significantly positively correlated with perceived dance quality. Linear regression subsequently revealed that three movement measures were key predictors of dance quality; these were variability and amplitude of movements of the neck and trunk, and speed of movements of the right knee. In summary, we have identified specific movements within men's dance that influence women's perceptions of dancing ability. We suggest that such movements may form honest signals of male quality in terms of health, vigour or strength, though this remains to be confirmed.

Meta-analysis of digit ratio 2D:4D shows greater sex difference in the right hand.

OBJECTIVES: Our aims are, first, to describe the sex difference in the length ratio of the second and fourth digits (2D:4D), which likely reflects prenatal testosterone levels in humans. Second, to infer the loss of reliability observed in 2D:4D based on self-measured finger lengths. METHODS: We used random-effects meta-analysis of 2D:4D based on expert-measured finger lengths (116 samples with 13,260 females and 11,789 males). RESULTS: We find a moderate sex difference (with lower 2D:4D for males), which shows substantial heterogeneity (which is unrelated to age). The sex difference is moderated by the type of finger length measurement and by hand. Measurement involving the distortion of soft tissue leads to a significantly larger sex difference than finger length measurement avoiding this. The sex difference in 2D:4D is larger in the right hand than in the left. The reliability of self-measured 2D:4D in the BBC internet study, by far the largest study on 2D:4D, is estimated to be 46% of that of expert-measured 2D:4D. CONCLUSIONS: Right-hand 2D:4D might be a better indicator of prenatal androgenisation than left-hand 2D:4D. The view that 2D:4D has allometric properties (Kratochvil L, Flegr J. 2009. Differences in 2nd to 4th digit length ratio in humans reflect shifts along the common allometric line. Biol Lett 5:643-646.) is not supported. Bone lengths contribute to the sex difference in 2D:4D. In addition, there might be a sex difference in fingers' soft tissue, which should be investigated. Because of measurement unreliability, correlations between 2D:4D and variables of interest are about one-third smaller in the BBC internet study than in studies in which 2D:4D is based on expert-measured finger lengths.

Progressive stage transition does mean getting better: a further test of the Transtheoretical Model in recovery from alcohol problems.

AIMS: To test two central assumptions of the Transtheoretical Model (TTM) regarding recovery from alcohol problems: (i) individuals making a forward transition from pre-action to action stages will show greater drinking improvements than those remaining in pre-action stages; and (ii) individuals remaining in pre-action stages will not demonstrate improvements in drinking outcomes. DESIGN AND SETTING: Large, multi-centre, randomized controlled trial of treatment for alcohol problems [United Kingdom Alcohol Treatment Trial (UKATT)]. MEASUREMENTS: Stage of change, drinks per drinking day and percentage days abstinent at baseline, 3- and 12-month follow-ups. FINDINGS: In support of TTM assumption 1, improvements in drinking outcomes were consistently greater among clients who showed a forward stage transition (Cohen's d = 0.68) than among those who did not (d = 0.10). Two tests of assumption 2 showed a significant improvement in drinking outcomes in non-transition groups, inconsistent with the TTM; one test showed a significant deterioration and the other showed equivalent drinking outcomes across time. An explanation is offered as to why, under the relevant assumption of the TTM, clients in non-transition groups showed small changes in drinking outcomes. CONCLUSIONS: In contrast to a previous study by Callaghan and colleagues, our findings largely support the TTM account of recovery from alcohol problems in treatment. The discrepancy can be explained by the use in our study of a more reliable and valid method for assigning stage of change.

Sometimes the impact factor outshines the H index.

Journal impact factor (which reflects a particular journal's quality) and H index (which reflects the number and quality of an author's publications) are two measures of research quality. It has been argued that the H index outperforms the impact factor for evaluation purposes. Using articles first-authored or last-authored by board members of Retrovirology, we show here that the reverse is true when the future success of an article is to be predicted. The H index proved unsuitable for this specific task because, surprisingly, an article's odds of becoming a 'hit' appear independent of the pre-eminence of its author. We discuss implications for the peer-review process.

Second to fourth digit length ratio (2D:4D) and adult sex hormone levels: new data and a meta-analytic review.

The relative length of the second (index) to the fourth (ring) finger (2D:4D) is a putative negative correlate of prenatal testosterone (T) exposure. Therefore, 2D:4D (and to a lesser extent D(r-l), the difference between 2D:4D in the right hand and in the left hand) has often been used to study effects of prenatal androgenization on human behavior and cognition. However, evidence suggests that 2D:4D may also be related to levels of circulating sex hormones in adults. This would question the validity of 2D:4D as a means of studying the effects of prenatal sex hormones. Here we present new data from two non-clinical samples (64 women and 102 men) regarding the relationships of 2D:4D and D(r-l) with circulating sex hormone levels. We then present a meta-analytic review of all the present evidence regarding this issue. The results suggest that, in the normal population, 2D:4D and D(r-l) are not associated with adult sex hormone levels. The findings from this current study add to the growing body of evidence demonstrating that 2D:4D is a suitable tool to study the effects of prenatal androgenization on human behavior and cognition.

The meaning and suitability of various effect sizes for structured Rater x Ratee designs.

Four types of analysis are commonly applied to data from structured Rater x Ratee designs. These types are characterized by the unit of analysis, which is either raters or ratees, and by the design used, which is either between-units or within-unit design. The 4 types of analysis are quite different, and therefore they give rise to effect sizes that differ in their substantive interpretations. In most cases, effect sizes based on between-ratee analysis have the least ambiguous meaning and will best serve the aims of meta-analysts and primary researchers. Effect sizes that arise from within-unit designs confound the strength of an effect with its homogeneity. Nonetheless, the authors identify how a range of effect-size types such as these serve the aims of meta-analysis appropriately.

Once more: is beauty in the eye of the beholder? Relative contributions of private and shared taste to judgments of facial attractiveness.

Misconstruing the meaning of Cronbach's alpha, experts on facial attractiveness have conveyed the impression that facial-attractiveness judgment standards are largely shared. This claim is unsubstantiated, because information necessary for deciding whether judgments of facial attractiveness are more influenced by commonly shared or by privately held evaluation standards is lacking. Three experiments, using diverse face and rater samples to investigate the relative contributions of private and shared taste to judgments of facial attractiveness, are reported. These experiments show that for a variety of ancillary conditions, and contrary to the prevalent notion in the literature, private taste is about as powerful as shared taste. Important implications for scientific research strategy and laypeople's self-esteem are discussed.

Digit ratio (2D:4D) and physical fitness in males and females: Evidence for effects of prenatal androgens on sexually selected traits.

It has been suggested that male achievement in sports and athletics is correlated with a putative measure of prenatal testosterone the 2nd to 4th digit ratio (2D:4D). It is not known whether this association also extends to females, or whether the association results from an effect of testosterone on behavior (such as exercise frequency) or on physical fitness. Here, we report for the first time data from two studies which consider associations between 2D:4D and physical fitness in females in addition to males: Study I--in a sample of teenage boys (n = 114) and girls (n = 175), their 'physical education grade' was negatively associated with 2D:4D of the right hand (boys), and right and left hand (girls), and Study II-among a sample of young men (n = 102) and women (n = 77), a composite measure of physical fitness was negatively related to right hand 2D:4D in men and left hand 2D:4D in women. We conclude that 2D:4D is negatively related to physical fitness in both men and women. In Study II, there was evidence that the relationship between physical fitness and 2D:4D in men was mediated through an association with exercise frequency. Thus, 2D:4D in males may be a negative correlate of frequent exercise which then relates to achievement in sports and athletics.

2nd to 4th digit ratio (2D:4D) and number of sex partners: evidence for effects of prenatal testosterone in men.

The number of sexual partners per individual (NSP) is an important component of sexual behaviour. Here, we report two studies concerning the relationship between a probable negative correlate of prenatal testosterone, the ratio of the length of 2nd and 4th digits (2D:4D), and NSP in men. The right hand 2D:4D ratio appears to be more strongly related to prenatal testosterone than does the left hand. Accordingly we found: (a) in a sample of 99 German heterosexual male undergraduates right hand 2D:4D (but not left hand 2D:4D) was significantly negatively associated with reported lifetime NSP. The relationship between NSP and 2D:4D was independent of free testosterone, but free testosterone also showed a weak positive association with NSP (b) in a sample of 79 heterosexual and 95 homosexual Austrian men we found a significant negative association between right hand 2D:4D (but not left hand 2D:4D) and reported NSP in past year for heterosexual but not for homosexual men. The association in heterosexuals was independent of age, years of education, occupation and relationship status. We conclude that male NSP is likely to be influenced by the long-term organisational effects of prenatal testosterone. The relationship between NSP and 2D:4D appears to be confined to heterosexual men.

Facial attractiveness, symmetry, and physical fitness in young women.

This study explores the evolutionary-based hypothesis that facial attractiveness (a guiding force in mate selection) is a cue for physical fitness (presumably an important contributor to mate value in ancestral times). Since fluctuating asymmetry, a measure of developmental stability, is known to be a valid cue for fitness in several biological domains, we scrutinized facial asymmetry as a potential mediator between attractiveness and fitness. In our sample of young women, facial beauty indeed indicated physical fitness. The relationships that pertained to asymmetry were in the expected direction. However, a closer analysis revealed that facial asymmetry did not mediate the relationship between fitness and attractiveness. Unexpected problems regarding the measurement of facial asymmetry are discussed.